Bi-directional audio generation device for speech quality assessment of telephony networks and speech codecs

ABSTRACT

A system for bi-directional quality testing of a telephony system or network using an audio generation device configured to automate Perceptual Evaluation of Speech Quality assessments and connect to a communication endpoint through computer telephony integration (CTI) instruction or system configuration to enable auto-answering of the communication endpoint, listen, record, and process signals and audio data, calculate Mean Opinion Scores (MOS), generate signals and audio for playback, compare files using a full reference algorithm, and store data with MOS results displayed in data naming structure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/371,191 titled “BI-DIRECTIONAL AUDIO GENERATION DEVICE FOR SPEECHQUALITY ASSESSMENT OF TELEPHONY NETWORKS AND SPEECH CODECS”, filed onDec. 6, 2016, the entire specification of which is incorporated hereinby reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Art

The disclosure relates to the field of telephony, and more particularlyto the field of quality of experience.

Discussion of the State of the Art

Audio system measurements are instrumental to operating and maintaininga reliable telephony network. Measurements may help in sizing andspecifying new equipment or monitoring or troubleshooting existingequipment. Audio system measurements typically model psychoacousticprinciples to assess the quality of a communication system as it maypertain to human hearing. In telephony systems, a Mean Opinion Score(MOS) provides a numerical indication of quality of speech as perceivedby the listening end of the communication, and while it is subjective,MOS tests have been standardized and are specified by the InternationalTelecommunications Union, ITU-T, under published recommendation P.800.

A Mean Opinion Score (MOS) is a type of quality measurement test thathas been used for decades in telephony to obtain a human user'sperceived quality of a voice call or audio file played over the giventelephony network or system. Initially, a Mean Opinion Score is recordedby each listener as an integer in a range of 1 to 5, where 1 is thepoorest perceived quality and 5 is the best perceived quality. A MeanOpinion Score is determined by computing the arithmetic mean of allscores recorded from the results of a set of standardized tests where anumber of listeners score the perceived audible quality of a series ofprescribed test sentences read aloud by both male and female voiceswithin a prescribed test environment, over the communications mediumbeing tested. A Mean Opinion Score is a subjective measurement process,as listeners need to sit together in a quiet room and score call qualityas they perceive it. Obtaining a Mean Opinion Score is time-consumingand expensive, as it requires involving multiple prospective or activeusers to accomplish the protocols as required by the standardizedtesting methods.

In voice over internet protocol (VoIP) telephony, a Mean Opinion Scoreprovides a numerical indication of perceived quality of a received mediaafter compression/decompression (codec) and/or transmission has takenplace. Measurement applications for wireless, VoIP, fixed and codectelephony systems have been developed and are calculated based onperformance of the IP network used within the system, as well asempirically assimilated for automated MOS results via implementation ofITU-T P.862.1, mapping of Perceptual Evaluation of Speech Quality (PESQ)scores to Mean Opinion Score (MOS) scale.

Perceptual Evaluation of Speech Quality (PESQ) is presented in ITU-Trecommendation P.862, and comprises a test methodology for assessment ofspeech quality as experienced by users of telephony systems or networks.Specifically, PESQ was developed to standardize and automate MeanOpinion Score tests by employing algorithms to analyze speech signals.PESQ utilizes true voice samples as test signals; use of other,non-spoken audible signals, such as tones or noise, may yieldunpredictable results. PESQ is a full-reference (FR) algorithm andanalyzes speech signal sample-by-sample after a sequential alignment ofcorresponding excerpts of reference and test signal. PESQ may be appliedto provide end-to-end (E2E) quality assessment scores for a network orfor individual network components.

A full reference (FR) algorithm, such as PESQ, uses an establishedreference signal for comparison purposes. It can compare each sample ofthe reference signal (upstream end) to each corresponding sample of thedegraded signal (downstream end). Full reference measurements provide ahigh level of accuracy and repeatability, but they may only be appliedfor dedicated tests in live networks, therefore, technicians, or moregenerally, users, must be available on both ends to initiate, receiveand complete the testing and quality assessment of the live networkand/or its components.

What is needed in the art is an automated electronic device which may beconnected to a telephony network component or a telephony system inorder to accommodate and perform Perceptual Evaluation of Speech Qualityassessments in an imitated live scenario such that an estimated MeanOpinion Score may be produced for a telephony system without need forlive user involvement on either the dispatching (upstream) end or thereceiving (downstream) end.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in apreferred embodiment of the invention, an automated electronic computingdevice which may be connected to a telephony network component such thatan estimated Mean Opinion Score may be produced without need for liveuser involvement.

According to a preferred embodiment of the invention, a system forbi-directional quality testing of a telephony system or network using anaudio generation device, comprising a computer comprising a processor, amemory, and a plurality of programming instructions stored in the memoryand operable on the processor, wherein the plurality of programminginstructions is configured to: connect to a communication endpoint (suchas a PC sound device or a telephone turret) on one end of a call; listenthrough an analog interface of the communication endpoint for knowncommands originating from a prescribed call generation device; recordsignals and audio presented during the call; process signals and audio;calculate Mean Opinion Scores for downstream audio, and generate anupstream reference audio to facilitate the calculation of Mean OpinionScore at the call generation device; generate signals and audio for useduring testing sequence; compare files for quality assessment; storefiles locally; and allow prescribed remote access via in-band audiosignaling or via an adjunct network, is disclosed.

According to another preferred embodiment of the invention, a method forbi-directional quality testing of a telephony system or network operatedby a call generation device by using an audio generation deviceconnected to a communication endpoint, the method comprising the stepsof: placing a call from the call generation device to the communicationendpoint within the telephony system or network; answering the callgeneration device's call automatically at the dedicated communicationendpoint; pausing and waiting by the call generation device, for aconfirmed alive tone to be received by the audio generation device;sending an alive tone from the audio generation device, and the callgeneration device detecting and recording receipt of the alive tone;playing a specific tone from the call generation device to request theaudio generation device to play a specific, prescribed reference audiofile; preparing a local copy of the prescribed reference audio fileusing the audio generation device, then playing the reference audio fileaudibly as an utterance; capturing the utterance on the call generationdevice and storing the utterance as an audio file; processing, using thecall generation device, the captured utterance with a PerceptualEvaluation of Speech Quality Full Reference algorithm to calculate aMean Opinion Score; and recording and logging of the Mean Opinion Scoreby the call generation device, then continues processing further actionson the call until the endpoint disconnects the call, is disclosed.

According to another preferred embodiment of the invention, a method forbi-directional quality testing of a telephony system or network operatedby a call generation device by using an audio generation deviceconnected to a communication endpoint, the method comprising the stepsof: placing a call from the call generation device to the communicationendpoint within the telephony system or network; answering the callgeneration device's call automatically at the dedicated communicationendpoint; pausing and waiting by the call generation device, for aconfirmed alive tone to be received by the audio generation device;sending an alive tone from the audio generation device, and the callgeneration device confirming receipt of the alive tone; playing aspecific tone from the call generation device, to direct the audiogeneration device to begin capturing an audible utterance; playing aprescribed reference audio file from the call generation device, to becaptured as an utterance by the audio generation device; playing aspecific tone from the call generation device, to direct the audiogeneration device to stop capturing the audible utterance and store itas an audio file; processing, using the audio generation device, thecaptured utterance with a Perceptual Evaluation of Speech Quality FullReference algorithm to calculate a Mean Opinion Score; communicating theMean Opinion Score result from the audio generation device back to thecall generation device by way of audible speech; the call generationdevice recognizing the speech and logging the Mean Opinion Score;replaying the captured utterance file from the audio generation deviceback to the call generation device; and recording and logging of theutterance audio file by the call generation device, then continuesprocessing further actions on the call until the endpoint disconnectsthe call, is disclosed.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. It will beappreciated by one skilled in the art that the particular embodimentsillustrated in the drawings are merely exemplary, and are not to beconsidered as limiting of the scope of the invention or the claimsherein in any way.

FIG. 1 is a block diagram illustrating an exemplary architecture for atelephony system that uses a call generation device, a call deliveryframework, a communication endpoint for receiving a call, and an audiogeneration device connected to the communication endpoint to facilitateautomated testing protocols and sequences, according to a preferredembodiment of the invention.

FIG. 2 is a process flow diagram of a method for determining a MeanOpinion Score for a call experience, using a system of the invention.

FIG. 3 is another process flow diagram of a method for determining aMean Opinion Score for a call experience, using a system of theinvention.

FIG. 4 is an exemplary state transition diagram illustrating a pluralityof events that may occur in one or more possible stages during adownstream test sequence, according to a preferred embodiment of theinvention.

FIG. 5 is an exemplary state transition diagram illustrating a pluralityof events that may occur in one or more possible stages during anupstream test sequence, according to a preferred embodiment of theinvention.

FIG. 6 is a process flow diagram of a method that may be used to set andcheck necessary settings and conditions prior to completing the methodas depicted in FIG. 2 or the method as depicted in FIG. 3.

FIG. 7 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in an embodiment of theinvention.

FIG. 8 is a block diagram illustrating an exemplary logical architecturefor a client device, according to an embodiment of the invention.

FIG. 9 is a block diagram showing an exemplary architectural arrangementof clients, servers, and external services, according to an embodimentof the invention.

FIG. 10 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, in a preferredembodiment of the invention, an automated electronic computing devicewhich may be connected to a telephony network component such that anestimated Mean Opinion Score may be produced without need for live userinvolvement.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beappreciated that these are presented for illustrative purposes only andare not limiting of the inventions contained herein or the claimspresented herein in any way. One or more of the inventions may be widelyapplicable to numerous embodiments, as may be readily apparent from thedisclosure. In general, embodiments are described in sufficient detailto enable those skilled in the art to practice one or more of theinventions, and it should be appreciated that other embodiments may beutilized and that structural, logical, software, electrical and otherchanges may be made without departing from the scope of the particularinventions. Accordingly, one skilled in the art will recognize that oneor more of the inventions may be practiced with various modificationsand alterations. Particular features of one or more of the inventionsdescribed herein may be described with reference to one or moreparticular embodiments or figures that form a part of the presentdisclosure, and in which are shown, by way of illustration, specificembodiments of one or more of the inventions. It should be appreciated,however, that such features are not limited to usage in the one or moreparticular embodiments or figures with reference to which they aredescribed. The present disclosure is neither a literal description ofall embodiments of one or more of the inventions nor a listing offeatures of one or more of the inventions that must be present in allembodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or morecommunication means or intermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring non-simultaneously (e.g., because one step is described afterthe other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to one ormore of the invention(s), and does not imply that the illustratedprocess is preferred. Also, steps are generally described once perembodiment, but this does not mean they must occur once, or that theymay only occur once each time a process, method, or algorithm is carriedout or executed. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described herein, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described herein, it will be readily apparent that a singledevice or article may be used in place of the more than one device orarticle.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should beappreciated that particular embodiments may include multiple iterationsof a technique or multiple instantiations of a mechanism unless notedotherwise. Process descriptions or blocks in figures should beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process. Alternate implementations areincluded within the scope of embodiments of the present invention inwhich, for example, functions may be executed out of order from thatshown or discussed, including substantially concurrently or in reverseorder, depending on the functionality involved, as would be understoodby those having ordinary skill in the art.

Conceptual Architecture

FIG. 1 is a block diagram illustrating an exemplary architecture for atelephony system 100 that uses a call generation device 110 with accessto a database 120, a call delivery framework 140, a communicationendpoint 150 for receiving a call, such as, for example, a computingsound device or a telephone turret, an audio generation device 170connected to communication endpoint 150 via a specially made connection160 such that testing protocols and sequences may be automaticallyexecuted in a live environment, to test the call delivery framework 140or associated components in a simulated call originating from either endof the telephony system 100, according to a preferred embodiment of theinvention. The call generation device 110 may place a call over the calldelivery framework 140 to the audio generation device 170 by initiatingprotocols to direct the communication endpoint 150 to automaticallyanswer and connect the call to the audio generation device 170.Conversely, the audio generation device 170 may place a call to the callgeneration device 110 by initiating protocols to direct thecommunication endpoint 150 to initiate a call within the call deliveryframework 140, directed to the call generation device 110. The audiogeneration device 170 may be connected to system 100 to facilitate anautomatic Mean Opinion Score calculation. The audio generation device170 may be connected to a local administrative network 180, which maybe, for example, an in-band audio signaling network for basic functionssuch as device reboot or an adjunct IP network (wired Ethernet or WiFi)established for administrative purposes or onsite troubleshooting usingadministrative tools such as computing devices 190A/B/C/D. Typically,administrative tools 190A/B/C/D may not be accommodated on a networksupporting the call delivery framework 140, hence may need a separatenetwork 180 to access the audio generation device 170 without needing toobtain access permissions for the call delivery framework 140.

It should be appreciated that according to the embodiment, various meansof connection or communication between the components of system 100 maybe utilized according to the invention interchangeably orsimultaneously, such as for example a direct, physical data connection(such as via a data cable or similar physical means), a software-basedconnection such as via an application programming interface (API) orother software communication means (such as may be suitable, forexample, in arrangements where multiple system components may operate ona single hardware device such as a computing server or workstation), orany of a variety of network connections such as via the Internet orother data communications network. It should therefore be appreciatedthat the connections shown are exemplary in nature and represent only aselection of possible arrangements and that alternate or additionalconnections may be utilized according to the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 2 is process flow diagram illustrating an exemplary method 200 fortesting a system 100 in both directions: upstream from audio generationdevice 170 to call generation device 110, and downstream from callgeneration device 110 to audio generation device 170, in both cases, viaa dedicated communication endpoint 150 connected to a call deliveryframework 140. In a preferred embodiment of the invention, callgeneration device 110 sends known commands to audio generation device170 by playing tones of differing frequencies to identify and executecommands, such as changing volume settings of communication endpoint150, preparing audio generation device 170 to capture an audio file,processing an audio file to calculate a quality score, and/or initiatinga soft or hard reboot of methods 200/300. Audio generation device 170may be configured to play an alive tone 425/525 at predeterminedintervals, for example, every 10 seconds, to alert call generationdevice 110 that audio generation device 170 is available, in an idleposition, awaiting commands. In a preferred embodiment of the invention,call generation device 110, connected to system 100, places a call tocommunication endpoint 201, communication endpoint answers callautomatically 203, waits for alive tone 205 and if no tone is received,may terminate the call 210. In such a case, the process may start again.Once an alive tone 425/525 is confirmed 212, call generation device 110requests audio generation device to play 215 a predetermined referenceaudio file 469 which matches one stored in database 120. Reference audiofile 469 is stored on audio generation device 170, which prepares a copyof audio file 469 then plays audio file 469 as an utterance 220. Callgeneration device 110 receives, captures, and stores 230 reference audiofile 469 to process by comparing the transmitted audio file 469 to itsmatching counterpart as stored in database 120. The call generationdevice 110 processes the captured reference audio file 469 as anutterance 240 such that an experience score as a Mean Opinion Score(MOS) may be calculated 250 by the call generation device 110, inaccordance with a prescribed full reference algorithm as a function ofPESQ score mapping. Call generation device 110 records and logs a MeanOpinion Score 260, and may store it in database 120 before terminatingthe call 270.

FIG. 3 is another process flow diagram illustrating an exemplary method300 for testing a system 100 in both directions: upstream from audiogeneration device 170 to call generation device 110, and downstream fromcall generation device 110 to audio generation device 170, in bothcases, via a dedicated communication endpoint 150 connected to a calldelivery framework 140. In a preferred embodiment of the invention, callgeneration device 110 places a call 301 to communication endpoint 150,which automatically answers the call 303, waits for alive tone 305 andif no tone is received, may terminate the call 310. In such a case, theprocess may start again. Once an alive tone 425/525 is confirmed 312,call generation device 110 directs audio generation device 170 tocapture an audible utterance 315. Call generation retrieves audio filefrom database 120, plays audio as an utterance 320, then audiogeneration device captures the utterance 330. Call generation devicedirects audio generation device to stop capturing audio and store thecaptured audio file 335, then to process the captured utterance 340 suchthat the audio generation device 170 may calculate a Mean Opinion Score(MOS) 350, in order to speak the result 355 back to the call generationdevice 110. The call generation device 110 recognizes the speech andlogs the MOS result 360 into database 120. Audio generation device 170replays the captured utterance 370 to the call generation device 110,and the call generation device 110 records and stores the utterance file380 in database 120, before terminating the call 390.

FIG. 4 is exemplary state transition diagram 400 illustrating aplurality of stages of a Mean Opinion Score determination in an upstreamdirection, meaning a perceived quality of experience from audiogeneration device 415 via telephony system 410 to call generation device405. Audio generation device 415 plays an alive tone and listens forcommands 420 at predetermined intervals. Even though a call may start401 by call generation device 405 for the purpose of measuring aninbound or upstream MOS, call generation device 405 establishes aconnection with audio generation device 415 before generating anoutbound call 430 from call generation device 405 to audio generationdevice 415, which is automatically answered on a dedicated communicationendpoint 435 connected within telephony system 410. As connection isestablished, call generation device 405 waits 440 for alive tone 425,confirms receipt 450 of alive tone 425, before requesting audiogeneration device 415 to play a predetermined reference audio file 460in order to compare original file quality and respective audibleutterance to compare degradation. The request to play a predeterminedaudio file 460 originates as a tone 465 which signals audio generationdevice 415 to retrieve a local copy of the predetermined audio file 460from its local memory, and prepare local copy of requested referenceaudio file 467 for playback. Audio generation device 415 audibly playslocal copy of reference audio file 468 over telephony system 410 as anaudible file 469. Call generation device 405 captures audio file 469 asit would have been heard, an utterance 470, processes captured utteranceusing PESQ algorithm to calculate Mean Opinion Score 480, then recordsand logs MOS result 490 before terminating the call 499.

FIG. 5 is another exemplary state transition diagram 500 illustrating aplurality of stages of a Mean Opinion Score determination in adownstream direction, meaning a perceived quality of experience fromcall generation device 505 via telephony system 510 to audio generationdevice 515. Audio generation device 515 plays an alive tone and listensfor commands 520 at predetermined intervals. Even though a call maystart 501 by call generation device 505 for the purpose of measuring anoutbound or downstream MOS, call generation device 505 establishes aconnection with audio generation device 515 before generating anoutbound call 530 from call generation device 505 to audio generationdevice 515, which is automatically answered 535 on a dedicatedcommunication endpoint connected within telephony system 510. Asconnection is established, call generation device 505 waits 540 foralive tone 525, confirms receipt 550, and by playing a tone 556, alertsaudio generation device to capture an audible utterance 555. Uponreceipt of tone 556, audio generation device 515 begins capturingutterance 557 while call generation device 505 plays reference audiofile 560 as an audible speech-like sound 561. Audio generation device515 captures audio file 561 as an utterance 562 then processes thecaptured utterance file 562 to calculate a Mean Opinion Score using PESQalgorithm 567. Audio generation device 515 communicates the MOS resultwith check digit twice 570 by playing an audible speech file 575speaking the MOS results back to call generation device 505, whereresults are recognized and logged 580 by the call generation device 505.Audio generation device 515 replays captured utterance file 585 as anaudio file 590 back to call generation device 505 which stores 595utterance file 590 before terminating the call 599. Call generationdevice 110 may wait for audio generation device 170 to become idlebefore alerting it to capture an audible utterance. Call generationdevice 110 may wait for a plurality of time intervals, which in thisparticular example, may equate to multiples of ten seconds, to confirmavailability. When audio generation device 170 confirms itsavailability, call generation device 110 plays a reference file 561 thenalerts audio generation device 170 to stop capturing and store audiofile 561 as an utterance file 562. Audio generation device 170 sends 265utterance file 562 back upstream to call generation device 110, which iswaiting 270 until it is received to compare files 561 to 562 tocalculate a Mean Opinion Score.

Tones and remote commands executed between call generation device110/405/505 and audio generation device 170/415/515 operate in a seriesof frequencies, with each frequency tone representing a specificcommand, as detailed in table 1100 some key functions include asecondary (lower) tone frequency in case of traversing telephony devicesthat attempt to filter high frequency tones (shriek rejection):

Frequency Mark Command Line (Hertz) Action 1101 REQUEST-GET VOLUME1400/550 AGD responds with current volume level of AGD 1102REQUEST-VOLUME-UP 1300/600 AGD increases persistent volume by oneincrement 1103 REQUEST-VOLUME-DOWN 1350/650 AGD decreases persistentvolume by one increment 1104 PREPARE-FOR-REFERENCE 1000/500 AGD preparesto capture then process against a default, predetermined audio referencefile (reference.wav) 1105 PREPARE-FOR-REFERENCE-A 1050 AGD prepares tocapture then process against an alternate audio reference file(reference-alt-a.wav) 1106 PREPARE-FOR-REFERENCE-B 1075 AGD prepares tocapture then process against an alternate audio reference file(reference-alt-b.wav) 1107 PREPARE-FOR-REFERENCE-C 1150 AGD prepares tocapture then process against an alternate audio reference file(reference-alt-c.wav) 1108 PREPARE-FOR-REFERENCE-D 1200 AGD prepares tocapture then process against an alternate audio reference file(reference-alt-d.wav) 1109 PREPARE-FOR-REFERENCE-E 2045 AGD prepares tocapture then process against an alternate audio reference file(reference-alt-e.wav) 1110 END-OF-REFERENCE 1500/800 AGD stops capturingdegraded audio, processes captured utterance, responds with MOS result,replays captured audio. 1111 PLAY-REFERENCE 1750/750 AGD plays defaultreference audio file (reference.wav) to be captured and processed fordegradation by call generation device. 1112 PLAY-REFERENCE-A 1800 AGDplays an alternate reference audio (reference-alt-a.wav) to be capturedand processed for degradation by call generation device. 1113PLAY-REFERENCE-B 1850 AGD plays an alternate reference audio(reference-alt-b.wav) to be captured and processed for degradation bycall generation device. 1114 PLAY-REFERENCE-C 1900 AGD plays analternate reference audio (reference-alt-c.wav) to be captured andprocessed for degradation by call generation device. 1115PLAY-REFERENCE-D 1950 AGD plays an alternate reference audio(reference-alt-d.wav) to be captured and processed for degradation bycall generation device. 1116 PLAY-REFERENCE-E 2000 AGD plays analternate reference audio (reference-alt-e.wav) to be captured andprocessed for degradation by call generation device. 1117REQUEST-REBOOT-SOFT 2100 For administration purposes only, AGD willattempt to close running processes and reboot. 1118 REQUEST-REBOOT-HARD2150 For administration purposes only, AGD will immediately reboot.

FIG. 6 illustrates a method 600 which may be inserted into either method200 or method 300, interjected between method steps 212 and 215 or 312and 315, respectively, and in this example, allows for volume settingsand signal power adjustments to be made prior to proceeding to eitherstep 215 or 315. It is crucial that volume settings on bothcommunication endpoint 150 and the connected audio generation device 170are aligned and balanced in order to obtain accurate MOS results. Ifvolume settings are not balanced between communication endpoint 150 andaudio generation device 170, skewed MOS results may occur. Accordingly,the inventor has conceived and reduced to practice, in a preferredembodiment of the invention, a prescribed list of settings for specificcommunication endpoint 150 makes and models, along with theircorresponding preferred volume settings for the connected audiogeneration device 170. In an instance where call generation device 110determines that volume settings may be imbalanced, method 600 may beexecuted. Call generation device 110 waits 205/305 to receive an alivetone 425/525 from audio generation device 170 before requestingcommunication endpoint volume result 605. Audio generation device 170plays 610 result back to call generation device 110, which recordssignal power 615. Call generation device 110 waits for audio generationdevice 170 to become idle 620 before requesting audio generation device170 volume level 625. In response, audio generation device 170 playsvolume level 630 back to call generation device 110 then transitions toan idle state 635 whereby the elected process method resumes 640. Eithermethod 200 resumes via method step 215, or method 300 resumes via methodstep 315.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be describedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, or other appropriatecomputing device), a consumer electronic device, a music player, or anyother suitable electronic device, router, switch, or other suitabledevice, or any combination thereof. In at least some embodiments, atleast some of the features or functionalities of the various embodimentsdisclosed herein may be implemented in one or more virtualized computingenvironments (e.g., network computing clouds, virtual machines hosted onone or more physical computing machines, or other appropriate virtualenvironments).

Referring now to FIG. 7, there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one embodiment, a computing device 10 may beconfigured or designed to function as a server system utilizing CPU 12,local memory 11 and/or remote memory 16, and interface(s) 15. In atleast one embodiment, CPU 12 may be caused to perform one or more of thedifferent types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 13 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a specific embodiment,a local memory 11 (such as non-volatile random access memory (RAM)and/or read-only memory (ROM), including for example one or more levelsof cached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QualcommSNAPDRAGON

or Samsung EXYNOS

CPU as are becoming increasingly common in the art, such as for use inmobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE

, THUNDERBOLT

, PCI, parallel, radio frequency (RF), BLUETOOTH

, near-field communications (e.g., using near-field magnetics), 802.11(WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces, GigabitEthernet interfaces, Serial ATA (SATA) or external SATA (ESATA)interfaces, high-definition multimedia interface (HDMI), digital visualinterface (DVI), analog or digital audio interfaces, asynchronoustransfer mode (ATM) interfaces, high-speed serial interface (HSSI)interfaces, Point of Sale (POS) interfaces, fiber data distributedinterfaces (FDDIs), and the like. Generally, such interfaces 15 mayinclude physical ports appropriate for communication with appropriatemedia. In some cases, they may also include an independent processor(such as a dedicated audio or video processor, as is common in the artfor high-fidelity A/V hardware interfaces) and, in some instances,volatile and/or non-volatile memory (e.g., RAM).

Although the system shown and described above illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 13 handles communicationsas well as routing computations, while in other embodiments a separatededicated communications processor may be provided. In variousembodiments, different types of features or functionalities may beimplemented in a system according to the invention that includes aclient device (such as a tablet device or smartphone running clientsoftware) and server systems (such as a server system described in moredetail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 16 and local memory 11) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 16 or memories 11,16 may also be configured to store data structures, configuration data,encryption data, historical system operations information, or any otherspecific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA

compiler and may be executed using a Java virtual machine or equivalent,or files containing higher level code that may be executed by thecomputer using an interpreter (for example, scripts written in Python,Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 8,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 20 includes processors 21that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 24. Processors 21 may carry out computinginstructions under control of an operating system 22 such as, forexample, a version of Microsoft's WINDOWS

operating system, Apple's Mac OS/X or iOS operating systems, somevariety of the Linux operating system, Google's ANDROID

operating system, or the like. In many cases, one or more sharedservices 23 may be operable in system 20, and may be useful forproviding common services to client applications 24. Services 23 may forexample be WINDOWS

services, user-space common services in a Linux environment, or anyother type of common service architecture used with operating system 21.Input devices 28 may be of any type suitable for receiving user input,including for example a keyboard, touchscreen, microphone (for example,for voice input), mouse, touchpad, trackball, or any combinationthereof. Output devices 27 may be of any type suitable for providingoutput to one or more users, whether remote or local to system 20, andmay include for example one or more screens for visual output, speakers,printers, or any combination thereof. Memory 25 may be random-accessmemory having any structure and architecture known in the art, for useby processors 21, for example to run software. Storage devices 26 may beany magnetic, optical, mechanical, memristor, or electrical storagedevice for storage of data in digital form (such as those describedabove). Examples of storage devices 26 include flash memory, magnetichard drive, CD-ROM, and/or the like.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers.

Referring now to FIG. 9, there is shown a block diagram depicting anexemplary architecture 30 for implementing at least a portion of asystem according to an embodiment of the invention on a distributedcomputing network. According to the embodiment, any number of clients 33may be provided. Each client 33 may run software for implementingclient-side portions of the present invention; clients may comprise asystem 20 such as that illustrated above. In addition, any number ofservers 32 may be provided for handling requests received from one ormore clients 33. Clients 33 and servers 32 may communicate with oneanother via one or more electronic networks 31, which may be in variousembodiments any of the Internet, a wide area network, a mobile telephonynetwork (such as CDMA or GSM cellular networks), a wireless network(such as WiFi, Wimax, LTE, and so forth), or a local area network (orindeed any network topology known in the art; the invention does notprefer any one network topology over any other). Networks 31 may beimplemented using any known network protocols, including for examplewired and/or wireless protocols.

In addition, in some embodiments, servers 32 may call external services37 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 37 may take place, for example, via one or morenetworks 31. In various embodiments, external services 37 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 24 are implemented on a smartphone or other electronicdevice, client applications 24 may obtain information stored in a serversystem 32 in the cloud or on an external service 37 deployed on one ormore of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 33 or servers 32 (or both)may make use of one or more specialized services or appliances that maybe deployed locally or remotely across one or more networks 31. Forexample, one or more databases 34 may be used or referred to by one ormore embodiments of the invention. It should be understood by one havingordinary skill in the art that databases 34 may be arranged in a widevariety of architectures and using a wide variety of data access andmanipulation means. For example, in various embodiments one or moredatabases 34 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, Hadoop Cassandra, Google BigTable, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 36 and configuration systems 35. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 36 orconfiguration system 35 or approach is specifically required by thedescription of any specific embodiment.

FIG. 10 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to keyboard 49, pointing device 50,hard disk 52, and real-time clock 51. NIC 53 connects to network 54,which may be the Internet or a local network, which local network may ormay not have connections to the Internet. Also shown as part of system40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

What is claimed is:
 1. A system for bi-directional quality testing of atelephony system or network using an audio generation device,comprising: a computer comprising a processor, a memory, and a pluralityof programming instructions stored in the memory and operable on theprocessor, wherein the plurality of programming instructions, whenoperating on the processor, cause the processor to: monitor an analogaudio interface of a communication endpoint for known commands andreference audio originating from a call generation device; recordsignals and audio received during a call at the analog audio interface;compute voice quality for speech audio received at the analog audiointerface; generate reference audio to facilitate calculation of a meanopinion score at the call generation device; generate, and transmitusing the call generation device via the call, signals and audio for useduring a testing sequence; and allow remote access via in-band audiosignaling or via an adjunct network.
 2. A method for bi-directionalquality testing of a telephony system or network operated by a callgeneration device by using an audio generation device connected to acommunication endpoint, the method comprising the steps of: (a) placinga call from the call generation device to the communication endpointwithin the telephony system or network; (b) answering the callautomatically at the communication endpoint; (c) receiving a confirmedalive tone at the audio generation device; (d) sending an alive tonefrom the audio generation device; (e) playing a specific tone from thecall generation device to request the audio generation device to play aspecific, prescribed reference audio file; (f) preparing a local copy ofthe prescribed reference audio file using the audio generation device,then playing the reference audio file as an utterance; (g) capturing theutterance on the call generation device and storing the utterance as anaudio file; (h) processing, using the call generation device, thecaptured utterance to calculate an indicia of voice quality; and (i)recording the indicia of voice quality by the call generation device,then terminating the call.
 3. The system of claim 1, wherein the signalsand audio are received from the analog audio interface simultaneouslywith the transmission of signals and audio for use during a testingsequence using the call generation device.
 4. The system of claim 1,wherein the computer is further configured to transmit, via the callgeneration device, the results of a mean opinion score calculation. 5.The system of claim 1, wherein the adjunct network is an InternetProtocol-based network.
 6. The system of claim 1, wherein the signalsreceived comprise at least an alive tone indicating a remote audiogeneration device is awaiting commands.
 7. The system of claim 1,wherein the signals transmitted comprise at least an alive toneindicating the call generation device is awaiting commands.
 8. Thesystem of claim 6, wherein the alive tone is received at regularintervals during a call.
 9. The system of claim 8, wherein the call isterminated if a predetermined number of intervals pass without receivingan alive tone.
 10. The method of claim 2, wherein the indicia of voicequality is a mean opinion score.
 11. The method of claim 7, furthercomprising the step of transmitting, via the call generation device, themean opinion score.
 12. The method of claim 7, wherein the signalsreceived comprise at least an alive tone indicating the call generationdevice is awaiting commands.
 13. The method of claim 7, wherein thesignals transmitted comprise at least an alive tone indicating the audiogeneration device is awaiting commands.
 14. The method of claim 12,wherein the alive tone is received at regular intervals during a call.15. The method of claim 14, wherein the call is terminated if apredetermined number of intervals pass without receiving an alive tone.